[LoRA] device_map fix when loading LoRAs #9827
Conversation
Co-authored-by: Benjamin Bossan <[email protected]>
Co-authored-by: Marc Sun <[email protected]>
Co-authored-by: Steven Liu <[email protected]>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| return ignore_patterns | ||
|
|
||
|
|
||
| def model_has_device_map(model): |
There was a problem hiding this comment.
This is the most important bit to take note of. And I have added tests for it too (test_sharded_components_can_be_device_placed()).
|
Failing tests are unrelated. |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
|
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
2 participants
What does this PR do?
Follow-up of #9449. Also clubs #9821.
In short, this enables distributed inference when using LoRAs.